build: pin dynamo-llm version in backend-common for cargo publish - #13303

🔴 Nightly builds will fail because a hardcoded version number is not updated by the version-stamping step

The shared backend crate now hardcodes the required version of the LLM crate (version = "1.4.0" at lib/backend-common/Cargo.toml:26) in a file that the nightly version-stamping step never rewrites, so once nightly renames every crate version to a dated pre-release the requirement no longer matches and the build stops.
Impact: Nightly container/wheel builds break with a dependency version mismatch, and every future release-version bump silently leaves this pin stale.

lib/backend-common/Cargo.toml now pins its dynamo-llm path dependency with a literal `version = "1.4.0"` so that `cargo publish` accepts it. However .github/scripts/apply_dev_version.py, which nightly CI runs (see .github/workflows/shared-build-image.yml), only rewrites versions in the root Cargo.toml (including internal path-dep pins) and in the explicit SUBCRATE_CARGO_TARGETS list; lib/backend-common/Cargo.toml is in neither, and its dep pin uses the inline-table form that the line-anchored VERSION_LINE_RE does not match. Consequence: after nightly stamps lib/llm to 1.4.0-devYYYYMMDD, backend-common still requires ^1.4.0, which a pre-release does not satisfy, and cargo fails to resolve the workspace. Consider either teaching apply_dev_version.py to rewrite internal path-dep version pins in arbitrary sub-crate Cargo.tomls (including inline tables) and registering lib/backend-common/Cargo.toml, or avoiding a hardcoded literal here (e.g. inherit from the workspace dependency entry and turn off default features some other way, such as a workspace-level dynamo-llm entry with default-features = false, or a feature-gated variant) so there is a single source of truth for the version. Also add a check that any internal path-dep version pin equals [workspace.package].version so future release bumps can't leave it stale.

Was this helpful? React with 👍 or 👎 to provide feedback.

Good catch — confirmed and fixed in 907976f.

Reproduced exactly: apply_dev_version.py .dev20260814 stamps the workspace to 1.4.0-dev20260814 while this pin stays ^1.4.0, and resolution fails:

error: failed to select a version for the requirement `dynamo-llm = "^1.4.0"`
candidate versions found which didn't match: 1.4.0-dev20260814

One correction to the suggested remedy: adding this file to SUBCRATE_CARGO_TARGETS would not fix it — VERSION_LINE_RE is line-anchored and deliberately skips inline dep tables (name = { version = "..." }), so the pin would survive stamping untouched. Instead, 907976f gives rewrite_root_cargo a WORKSPACE_PIN_CARGO_TARGETS list of member manifests that pin the workspace version inline, and applies the same exact-string rewrite it already uses for the root path-dep pins. Verified on a scratch checkout: after stamping, the pin follows the workspace version, full cargo metadata resolution passes, and re-running the stamper stays a no-op.

Why the pin exists at all: cargo publish rejects a path dep without a version (this is why dynamo-backend-common has missed every crates.io release since 1.2.1 — the old publish loop masked the error), and workspace = true cannot be combined with default-features = false (cargo: `default-features = false` cannot override workspace's `default-features`), so a versioned path dep is the only representable form.

P1 — reopening with evidence: the mechanism in 907976f is right, but it no longer engages on this PR's own base branch.

rewrite_root_cargo builds pin_re from the root's [workspace.package].version literal and then applies that exact string to WORKSPACE_PIN_CARGO_TARGETS. That is correct provided the pin here equals the workspace version. It no longer does: main moved to 1.5.0 (Cargo.toml:48) while this line still says 1.4.0, so the new loop matches nothing here and skips the write without a word.

Ran the stamper on a pristine origin/main tree with this PR's two files applied:

$ python3 .github/scripts/apply_dev_version.py ".dev20260915" .
apply_dev_version: stamped suffix '.dev20260915'
root [workspace.package].version: version = "1.5.0-dev20260915"
root dynamo-llm ws dep:          dynamo-llm = { path = "lib/llm", version = "1.5.0-dev20260915" }
backend-common dynamo-llm pin:   dynamo-llm = { path = "../llm", version = "1.4.0", default-features = false }

Exit 0. The result is the failure you reproduced, one minor version along — confirmed on a minimal workspace of the same shape:

error: failed to select a version for the requirement `llm = "^1.4.0"`
candidate versions found which didn't match: 1.5.0-dev20260915

Two controls, so this is scoped correctly:

• workspace left unstamped at 1.5.0 with the pin at ^1.4.0 resolves fine — caret accepts 1.5.0, so ordinary builds and the merge itself are unaffected. This bites the nightly path only.
• pin set equal to the workspace version and then stamped resolves fine, and a re-run is a no-op. The mechanism you added is sound; only the literal is wrong.

Pre-merge CI cannot catch it either way: the stamping step is gated if: inputs.dev_version_suffix != '' (shared-build-image.yml:487) and only nightly-ci.yml supplies a suffix, so the first signal would be a red nightly after merge.

Worth noting the release line already settled this convention: v1.4.1 pins 1.4.1 and v1.4.2 pins 1.4.2 — the literal tracks [workspace.package].version at every tag, the latter via a commit (ac4d289c) whose entire content is that bump.

Verified: with that value, stamping a clean origin/main tree carries the pin to 1.5.0-dev20260915 alongside the root, and re-running the stamper is a no-op.

One caveat this exposes, which is why I have also left a P2 on .github/scripts/apply_dev_version.py: nothing makes the stamper tell you the pin drifted. It treats a zero-match rewrite as success. Whatever value lands here will need the same manual bump at 1.6.0 unless the script starts failing loudly.

[P1] Still live, and the gap is now wider. lib/backend-common/Cargo.toml:26 pins 1.4.0, but main moved to 1.6.0 on 2026-09-17 (#15009). The new rewrite loop matches nothing here and skips the write without a word. My earlier 1.5.0 suggestion is stale too. Please set the pin to 1.6.0.

$ python3 .github/scripts/apply_dev_version.py ".dev20260918" .
apply_dev_version: stamped suffix '.dev20260918'
EXIT=0

root:           version = "1.6.0-dev20260918"
root ws dep:    dynamo-llm = { path = "lib/llm", version = "1.6.0-dev20260918" }
backend-common: dynamo-llm = { path = "../llm", version = "1.4.0", default-features = false }

error: failed to select a version for the requirement `llm = "^1.4.0"`
candidate versions found which didn't match: 1.6.0-dev20260918

P2 — this loop cannot distinguish "pin rewritten" from "pin not found".

pin_re is built from the root's [workspace.package].version literal, so when a member's pin has drifted from it, sub matches nothing, t2 == t, and the if simply skips the write. The script still prints apply_dev_version: stamped suffix ... and exits 0.

Verified on a clean origin/main tree with this PR's two files applied — the stamper exits 0 and leaves lib/backend-common/Cargo.toml at version = "1.4.0" while the root moves to 1.5.0-dev20260915:

$ python3 .github/scripts/apply_dev_version.py ".dev20260915" .
apply_dev_version: stamped suffix '.dev20260915'
root [workspace.package].version: version = "1.5.0-dev20260915"
root dynamo-llm ws dep:          dynamo-llm = { path = "lib/llm", version = "1.5.0-dev20260915" }
backend-common dynamo-llm pin:   dynamo-llm = { path = "../llm", version = "1.4.0", default-features = false }

The resulting cargo failure is on the lib/backend-common/Cargo.toml:26 thread.

This drift is not hypothetical: on the release line ac4d289c is a commit whose entire content is hand-bumping this same literal from 1.4.1 to 1.4.2. Nothing checks it, and pre-merge CI never runs this path (the step is gated if: inputs.dev_version_suffix != '' and only nightly-ci.yml supplies one), so the first signal is a red nightly.

A second, smaller point shapes the fix below: the root is written at line 157, before this loop runs. Any failure inside the loop therefore leaves the root stamped and the members not — and the "already stamped" early return at line 151 then makes a re-run a silent no-op rather than a recovery. Verified by removing the member manifest: the run dies with FileNotFoundError, the root is left at 1.5.0-dev20260915, and re-running after restoring the file prints success while leaving the pin at 1.4.0.

Validating every target before writing anything covers both:

Verified on fresh origin/main trees with this PR applied:

• stale 1.4.0 pin -> raises lib/backend-common/Cargo.toml: no version = "1.5.0" pin found -- it has drifted from [workspace.package].version and would not be stamped, and the root is left at 1.5.0 (no partial write);
• pin corrected to 1.5.0 -> exit 0, root and pin both 1.5.0-dev20260915;
• immediate re-run with the same suffix -> exit 0, still 1.5.0-dev20260915 (idempotent).

Unverified: I could not run the repo's black/ruff hooks — neither is installed on this machine — so the suggestion is syntax-checked (ast.parse) and behaviourally tested, but let pre-commit confirm the formatting.

A regression test would also have caught this cheaply; there is currently no test anywhere for apply_dev_version.py. Asserting "after stamping, every WORKSPACE_PIN_CARGO_TARGETS pin equals [workspace.package].version" fails on this PR's head as it stands.

[P2] Still live. The script still cannot report a skipped pin. .github/scripts/apply_dev_version.py:167. A no-match leaves the text unchanged, the if skips the write, and the run prints the same success line and exits 0 as a run that rewrote the pin. Please make a zero-match fail loudly. The suggestion on this thread still applies unchanged.

-k "apply_dev_version or dev_version or stamp"   no tests collected (900 deselected)
-k "router"                                      17/900 tests collected (883 deselected)

RuntimeError: lib/backend-common/Cargo.toml: no version = "1.6.0" pin found -- it has drifted from [workspace.package].version and would not be stamped